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ABSTRACT 


The purpose of this thesis is to analyze the probabilistic cost model currently in 
use by NAVSEA 05C to predict cost uncertainty in naval vessel construction and to 
develop a method that better predicts the ultimate cost risk. The data used to develop the 
improved approach is collected from analysis of the CG(X) class ship by NAVSEA 05C. 
The NAVSEA 05C cost risk factors are reviewed and analyzed to determine if different 
factors are better cost predictors. The impact of data elicitation, the Money Allocated Is 
Money Spent (MAIMS) principle, and correlation effects are incorporated into the 
research and analysis of this thesis. Data quality is directly affected by data elicitation 
methods and influences the choice of probability distribution used to give the best 
predictor of cost risk. MAIMS and correlation effects are shown to make a significant 
impact to the overall cost model. Program managers and analysts can readily implement 
the enhanced models using commercial Excel add-ins, such as Crystal Ball or ©Risk, and 
integrate them into their current cost risk analysis and management practices to better 
mitigate risk and control project cost. 
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EXECUTIVE SUMMARY 


In order to generate the funds to implement the 30-year plan of future ships and 
capabilities, the Navy must explore different options for cost savings. Fundamental to 
the success of complex projects, such as naval vessel construction, is the ability to 
control, manage, and communicate the status of the risk reduction effort throughout the 
development and production cycles (Kujawski & Angelis, 2007). It is recognized that the 
Navy and the shipbuilding industry need to change their technical and business 
shipbuilding strategies in order to achieve the goal of a future Fleet that balances both 
capability and affordability. Cost risk assessment and analysis is one tool that can be 
utilized to help recapitalize costs used in the ship acquisition and building process. 

This thesis analyzes the probabilistic cost model currently in use by Naval Sea 
Systems Command Cost Engineering and Industrial Analysis Division (NAVSEA 05C) 
to predict cost uncertainty in naval vessel construction and to develop a method that 
better predicts the ultimate cost risk. The NAVSEA 05C’s cost analysis model for the 
proposed new cruiser, CG(X), encompasses all aspects of cost for the entire Elect, 
including inflation and profit. The data used in the NAVSEA model were acquired from 
subject matter expert (SME) inquiry using three-point estimates of high, most likely, and 
low values. The Navy is placing great emphasis on producing the best product for each 
dollar spent. In order to ensure the continued acquisition of CG(X), it is important that 
realistic cost risk analysis be performed so that program managers can make 
informed decisions. 

The cost model elements investigated in this thesis include data elicitation 
methods, probability distribution function (PDE) choice, correlation effects, and Money 
Allocated is Money Spent (MAIMS) principle effects. The most significant impact is 
seen with MAIMS and data elicitation effects. PDE choice and correlation effects have 
lesser impact upon the cost model. 

Methods of data elicitation are explored and the use of a direct fractile assessment 
(DEA) is recommended for future use (Kujawski, Alvaro, and Edwards, 2004), although 
the research in this thesis did not involve data acquisition. To simulate the use of a DEA 
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methodology, three-parameter Weibull distributions are employed to account for 
uncertainty associated with SME estimation of data. A Weibull (10%, 50%, 90%) 
distribution is used to simulate a more optimistic view of the uncertainty of data, while a 
Weibull (20%, 50%, 80%) distribution models a more pessimistic view. 

The methodology in choosing different probability distribution functions and their 
applicability to the model is evaluated. Specifically, triangular, lognormal, and two 
variations of the three-parameter Weibull distribution are considered. Once enhanced 
models are established, program managers can implement them into their current cost 
risk analysis practice to mitigate risk and control project cost. 

Two types of correlation effects are considered and modeled in this thesis. The 
first is the correlation between the components of the radar suite, and the second is the 
correlation between all the components of the electronics suite. The radar suite is one of 
the systems that make up the electronics suite. The results suggest that the correlation 
effects are important for probability values midway between the mean and the extremes, 
but there is little difference for correlation coefficients beyond 0.5. Further investigations 
are recommended to quantify correlation effects. 

MAIMS modified probability distributions are used to show the significance of 
budget allocation levels (Kujawski, Alvaro & Edwards, 2004). These distributions reflect 
an empirically observed effect, namely, that once a budget is allocated, the project cost 
will most likely be at least equal to the amount allocated. As the MAIMS modification 
value increases, the overall distribution cost rises with increasing probability. 

Credibility and realism are two key cost risk assessment criteria. The use of 
improved methods, such as those investigated in this thesis, are especially significant for 
today’s Navy during a time of budget hardship. If the Navy’s plans for a 
313-ship Fleet are to become a reality, the incorporation of cost risk analysis into 
acquisition and shipbuilding management is imperative. Reliable cost assessments can 
help deliver projects on time, at a lower cost, with a higher probability of success. 
Effective training of personnel involved in cost assessment and continued efforts to 
improve existing cost models will help to improve the Navy’s current cost 
estimating process. 
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I. 


INTRODUCTION 


Admiral Gary Roughead stated in the Chief of Naval Operation’s (CNO) 
Guidance for 2007-2008 that: 

We manage risk. We will identify, analyze, mitigate and then 
accept risk, appreciating that we must always consider the risks in 
aggregate across the entire force. Zero risk is not achievable nor 
affordable. We must manage risk and move forward to accomplish the 
mission while safeguarding our people and infrastructure 
(Roughhead, 2007). 

Vice Admiral K. M. McCoy took this a step further in 2008, in a statement made 
on assuming the position of Commander, Naval Sea Systems Command: 

Our Common mission is to develop, deliver and maintain ships and 
systems on time and on cost for the Navy. To build an affordable future 
fleet, we will focus on reducing acquisition costs, including applying more 
risk-based decisions to specifications and requirements 
(McCoy, 2008). 

The United States Navy is living and functioning in an era of ever expanding 
technology, more stringent requirements, and a growing need for more ships and 
resources, all while working with a limited budget. These factors all lead to inherent cost 
growth in the projects that are developed to provide the Fleet with the capabilities that it 
needs. In order for the United States Navy to acquire and provide a full, 
state-of-the-art, 313-ship Navy by 2020, as stated in the fiscal year (FY) 2007 plan 
(Department of the Navy, 2006, it is imperative that methods allowing full capitalization 
of each dollar spent by the Navy are developed and implemented. 

In February 2006, the United States Navy presented its FY2007 plan, which 
outlines the objective of increasing the current 285-ship Fleet to 313 ships by 2020 
(Department of the Navy, 2006). By 2008, the Navy increased the estimate of its annual 
cost for the 30-year plan by about 44% in real terms, but it is still approximately 7% less 
than independent cost estimates conducted by the Congressional Budget Office 
(O’Rourke, 2008). This increase in estimated cost poses a problem for the overall 
funding of the shipbuilding strategy proposed by the Secretary of the Navy. The 
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credibility of the Navy’s estimates and the ability to fund its shipbuilding plans have been 
questioned by Congress and industry (Cavas, 2008b). 

In order to adequately generate the funds to implement the 30-year plan of future 
ships and capabilities, the Navy must explore different options for cost savings. 
Fundamental to the success of complex projects, such as naval vessel construction, is the 
ability to control, manage, and communicate the status of the risk reduction effort 
throughout the development and production cycles (Kujawski & Angelis, 2007). It is 
recognized that the Navy and shipbuilding industry need to change their technical and 
business shipbuilding strategies in order to achieve the goal of a future Fleet that balances 
both capability and affordability. Cost risk assessment and analysis is one tool that can 
be utilized to help recapitalize costs used in the ship acquisition and building process. 

A. BACKGROUND 

Risk analysis is an important component of the cost analysis of new vessels 
because actual costs will always have a probability of differing from the estimate. 
Several reasons account for the difference between the estimate and actual cost, which 
can include lack of knowledge about the future, errors associated with assumptions and 
cost estimating equations, historical data inconsistencies, and factors considered in 
making the estimate. The overall purpose of risk analysis is to quantify the potential for 
error (Government Accountability Office (GAO), 2007). In the case of a cost estimate it 
is the probability that the actual cost will exceed the cost estimate or the budget. This 
cost estimate allows for the assessment of risk of a given program. 

Cost overruns and growth are an enduring problem that is not new to the Navy. A 
1939 inquiry from Secretary of the Navy Ray Spear asks the question, “Why do naval 
vessels cost so much?” In the answer to this inquiry, the reasons given include increased 
progress in marine engineering and naval construction, increased horsepower in 
shipbuilding, improved quality of building materials, inflation, and the practice of paying 
full price for the best you can buy naturally increases costs. Spear (1939) states that, 
“care must be taken in approving estimates to make sure that they are reasonable and held 
to in the cost of production. When contracts are negotiated the question of costs should 

be investigated and a detailed knowledge of approximate costs obtained.” Just as it was 
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recognized by the Secretary of the Navy in 1939, it is recognized by today’s Navy 
leadership that cost estimation is an integral part of the ship acquisition process. 

The Defense Acquisition Acronyms and Terms (2003) defines risk as: 

A measure of the inability to achieve program objectives within defined 
cost and schedule constraints. Risk is associated with all aspects of the 
program, e.g., threat, technology, design processes, or Work Breakdown 
Structure (WBS) elements. It has two components, the probability of 
failing to achieve a particular outcome, and the consequences of failing to 
achieve that outcome. 

Risk analysis and management can be used to help program managers more 
effectively make acquisition decisions and allocate their resources by allowing for a 
better understanding of program risks. Risk management is a continuous process in the 
acquisition and development of naval vessels. 

The Naval Sea Systems Command, Cost Engineering and Industrial Analysis 
(NAVSEA 05C) introduced Cost Risk Analysis (CRA) into the Navy’s PR09 Planning, 
Programming, Budgeting, and Execution System (PPBES) to help assess vessel costs in 
terms of quantifiable risk. Cost Risk Assessment is defined as the process of quantifying 
the uncertainties associated with major acquisition programs. It therefore allows for 
informed decisions with an estimated level of confidence (McCarthy, 2008). 

Typical sources of cost uncertainty in naval vessel construction are 
(Deegan & Eields, 2007): 

• Estimating Methodology . Cost Estimation Relationships (CER); 
learning/rate/curve assumptions; and cost-reduction initiatives. 

• Economic/business Eactors . Rates-wages, overhead. General and 
Administrative (G & A), etc.; Vendor/supplier stability; Inflation indices; 
Multiyear assumptions. 

• Technical Eactors . Technology Readiness Eevel (TRE); design and 
manufacturing complexity; software complexity; part or 
technology obsolescence. 

• Schedule Eactors . Potential for schedule delays, how schedule risks 
impact costs. 
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• Program Specific Issues . Requirements change, quantity change, 
funding uncertainty. 

• Errors of Omission and Commission . Failure to account for rework. 

• Other Factors . Strikes, acts of nature, accidents. 

One of the key objectives of CRA is to enable better risk management, which will 
simultaneously reduce program costs and increase the probability of success. Cost 
estimating is recognized by NAVSEA 05C as an essential element of effective program 
management, required for realistic program planning and decision making. Risk analysis 
is important because the previous methodology of using point estimates is “precisely 
wrong” (Deegan, 2007a). Risk cannot be assessed with a point estimate, as it represents a 
single value that serves as a best guess for the parameter to be defined. Decision makers 
may not be able to completely understand the influence of different variables on cost with 
the use of a point estimate. Conversely, the use of risk analysis allows the decision 
maker to utilize their acquisition experience, while quantifying the qualitative aspects of 
acquisition scenarios. 

Point estimates are not an accurate method for predicting costs in shipbuilding 
because they do not properly account for problems that may be encountered in the 
acquisition process, as described above. They may be either overly optimistic or overly 
pessimistic. Optimistic point estimates ignore the potential risk and uncertainty in a 
project, which is necessary for management to make informed decisions. Immature 
technology, uncertain product design, schedule problems, and unforeseen events all have 
risk associated with their end product. Risk analysis is necessary in order to incorporate 
the effect of risk into the overall cost. Pessimistic point estimates assume worst scenarios 
and unlikely high costs. Quantitative risk analysis allows the cost estimator to assign a 
realistic range of costs around a point estimate, which provides decision makers with a 
level of confidence in achieving a credible cost. 

The NAVSEA Cost Estimating Handbook (2005, p. 3-1) states: 

Cost estimators must be proficient and aware of the financial management, 
performance measurement, schedule analysis, acquisition management, as 
well as the technical aspects of a program to support the cost estimating 
process effectively. 
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The NAVSEA Cost Estimating Process is comprised of three parts, which are 
further divided into 12 tasks. The three parts are: Develop Approach, Perform Estimate, 
and Brief Results. Eigure 1 shows the breakdown of the 12 tasks within the three parts. 
This thesis focuses on the Develop Approach and Perform Estimate parts of the cost 
estimating process. 



Task 1. Tnidate Estimate Task 7. Run Model dc Genetate Point Tastr4% Pioduce Final E«inMe 

TsukZ DevelopTechniolBueluie Estimace Ta^l. 

Tuk 3. Obtain WBS Task 8. Conduct Cost Risk Analyais & Task IZ Brief Reaula 

Task 4. Establish Estimate Assumprions Incotpoiate into Estimate 

Task 5. Select Cost Estimating Methods Ptdiminaty Estimate 

&TooU Review 

Task 6. CoUeaDaea 


Part 1 Part 2 Part 3 

Eigure 1. NAVSEA 12-Step Cost Estimating Process (Prom: NAVSEA Cost 

Estimating Handbook, 2005) 

This thesis addresses the first two parts of the cost estimating process depicted in 
Eigure 1—Develop Approach and Perform Estimate. Data collection is a task within the 
Develop Approach part of cost estimation and can be regarded as the most important part 
of risk analysis. Bad data will produce bad results, regardless of the subsequent analysis. 
Data elicitation is often done ad hoc; however, several reliable methods and sources are 
available for data collection. Data quality is critical to the success of the analysis and 
plays a significant part in the results generated for cost estimation. This thesis will 
discuss improved methods for data collection in order to obtain more reliable and 
standardized data from subject matter experts (SMEs). 

Risk analysts use probability distributions rather than point estimates to represent 
the possible outcomes of an event. There is a significant difference between a point 
estimate and a distribution, in that the distribution provides the full range of values with 
their associated probabilities, while the point estimate presents a single value. This 
allows program management to make budget decisions, based on desired confidence 
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levels. Quality may differ, based on the method of collection. Two methods commonly 
used for data collection include database queries and interviews of SMEs or stakeholders 
(Deegan & Fields, 2007). This thesis analyzes the current NAVSEA 05C Cruiser 
(CG(X)) probabilistic cost model including data elicitation. 

The direct fractile assessment (DFA) method provides one of the most reliable 
and least bias-prone procedures for eliciting uncertain quantities from SMEs (Kujawski 
et ah, 2004). Data elicitation from SMEs is innately uncertain; three findings from 
psychological experiments conducted by Alpert and Raiffia (1982) are: 

• A systematic bias toward overconfidence is common. 

• Extreme value judgment is poor. 

• Maximum and minimum values are vague terms. What do these terms 
really mean? 

Based on the findings of Alpert and Raiffia (1982), Kujawski et al. (2004) propose the 
following guidelines for data elicitation: 

• Ask SMEs to provide 10th, 50th, and 90th percentiles values for cost 
elements. Avoid extreme values, abstract measures such as the mean or 
standard deviation, or specific distribution functions. Allow for discussion 
and education of the SME in terms of bias when giving data figures. 

• Calibrate each set of percentiles to reflect individual and project specific 
considerations, both pessimistic and optimistic. For estimates that might 
be overly optimistic, a cost analyst might choose to shift a 90th percentile 
value to perhaps 80th or 75th percentiles. 

Tasks involved in the Performing Estimate depicted in Figure 1 are running the 
model and generating a point estimate or probability distribution, conducting a cost risk 
analysis, and conducting a preliminary estimate review. 

Traditionally, triangular distributions have been used in cost estimation models 
because of the simplicity in entering the required data. The triangular distribution 
requires minimum or low, most likely, and high or maximum values. Other commonly 
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used distributions include normal, lognormal, and uniform. Table 1 lists eight of the 
most common probability distributions used for cost estimation and uncertainty analysis. 
This thesis investigates different methods for data elicitation and selecting appropriate 
distributions. The effects of using different distributions on cost risk are evaluated 
and identified. 


Distribution 

Description 

Shape 


Typical application 

Lognormal 

A continuous distribution positively 
skewed with a limitless upper bound 
and known lower bound; skewed to 
the right to reflect the tendency toward 
higher cost. 

P 

1 _ 


To characterize uncertainty in 
nonlinear cost estimating 
relationships. 



Values 


Normal 

Used for outcomes likely to occur on 
either side of the average value; 
symmetric and continuous, allowing 
for negative costs and durations. In a 
normal distribution, about 68% of the 

ProDabiirty . 

* / 

/ 

/ 

To assess uncertainty with 
cost estimating methods; the 
standard deviation or standard 
error of the estimate is used to 
determine dispersion. 


values fall within one standard 
deviation of the mean. 


Vatoci 


Beta 

Similar to normal distribution but does 
not allow for negative cost or duration, 
this continuous distribution can be 
symmetric or skewed. 

Probabiirtv . 

* / 

/ 


To capture outcomes biased 
toward the tail ends of a range; 
often used with engineering 
data or analogy estimates. 




Valuer 


Triangular 

Characterized by three points—most 
likely, pessimistic, and optimistic 
values—can be skewed or symmetric 
and is easy to understand because it 
is intuitive. One drawback is the 

* / 
/ 

.5 

To express technical 
uncertainty, because it works 
for any system architecture or 
design; also used to determine 
schedule uncertainty. 


absoluteness of the end points. 


Values 


Uniform 

Has no peaks because all values, 
including highest and lowest possible 
values, are equally likely. 

Prooftbrirty 

4 

■- luaty ttfouonoui 

With engineering data or 
analogy estimates. 




Values 


Weibull 

Versatile, able to take on the 
characteristics of other distributions, 
based on the value of the shape 
parameter “b”—e.g,, Rayleigh and 
exponential distributions can be 

4 

! V , 


In life data and reliability 
analysis because it can mimic 
other distributions and its 
objective relationship to 
reliability modeling. 


derived from it.* 


Values 


Source: DOD, NASA, SCEA, and Industry, 


The Rayleigh and exponential distributions are a class of continuous probability distribution. 

Table 1. Common Probability Distributions used in cost estimating uncertainty analysis 

(From: GAO Cost Assessment Guide, 2007) 

The Money Allocated is Money Spent (MAIMS) principle is based on 
Parkinson’s Law, where “Work expands to fill the time allotted” and “padding schedule 
estimates directly contribute to cost overruns” (Augustine, 1997). In other words, it 
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suggests that there will be no cost underruns, and that the project will come in at or above 
the cost to which it is funded. Implementing the MAIMS principle in Monte Carlo 
simulations modifies the basic probability distribution functions (PDF) by setting any 
value less than the money allocation point equal to that money allocation value. There 
will be no costs associated with a value less than this money allocation point. Utilizing 
the MAIMS principle, the PDFs are modified to include a spike or delta function at an 
arbitrary point, which is assumed to be the “money allocation point,” corresponding to 
the dollar amount allocated to the program manager for the project and/or project 
cost elements. 

Correlation effects between elements are analyzed. Correlation accounts for 
interrelationships between cost elements. Data elements can either be negatively, 
neutrally, or positively correlated and can either exist among cost elements within a 
subsystem or between elements in different subsystems. For example, take into 
consideration the elements of a ship. Positive correlation arise when increases in weight, 
size, and number of weapons systems onboard result in an increase in acquisition and 
shipbuilding costs. An increase in the complexity of a weapon system further forces an 
increase in cost of other systems such as power, cooling, control. Analysis would be 
greatly simplified if analysts could assume that all elements are independent, or that all 
elements are dependent. Since neither statement is true, correct correlation between 
elements is necessary to provide the most accurate representation of cost. 

Many software programs are available for cost risk analysis. This thesis uses 
Crystal Ball® as an add-in to Microsoft Excel®, because of its ease of use and because it 
is the current program used by NAVSEA 05C. Crystal Ball® generates the Monte Carlo 
simulations that become the backbone of the cost risk analysis. A Monte Carlo 
simulation calculates multiple scenarios of a model by repetitively sampling values from 
the input variable distributions for each uncertain variable and then calculates the result. 
Depending on the complexity of the model, the analysis may take only a few moments or 
hours (Wittwer, 2004). The resulting cost distributions from Crystal Ball® provide the 
decision maker with powerful cost risk information. 
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A program built on a solid foundation of accurate cost estimating that effectively 
considers risks, combined with strong systems engineering and program management, 
gives the program a greater chanee of success. 

B. PURPOSE 

The purpose of this thesis is to analyze the probabilistic cost analysis approach 
that NAVSEA’s Cost Engineering and Industrial Analysis division (SEA 05C) currently 
uses to predict new naval vessel construction costs and to develop a method that better 
predicts the ultimate cost risk. This thesis uses data colleeted from analysis of the CG(X) 
class ship cost model. The model used to determine cost is reviewed and analyzed to 
determine what factors should be considered to produce more realistic cost estimates. 

C. RESEARCH QUESTIONS 

Several questions are addressed in this thesis. They include: 

• What is the realism of the cost models currently in use by NAVSEA 05C 
for the CG(X) class ship? 

• How does data elicitation impact the cost prediction? 

• How does the choice of distributions in the model affect the predictions of 
the cost outcome? 

• What is the impact of the MAIMS principle on the cost 
probability distributions? 

• How can the current CG(X) model be improved to provide more accurate 
models to prediet cost and risk? 

D. BENEFITS OF STUDY 

The benefit of this study is to provide an independent analysis of the cost model 
developed by NAVSEA 05C that is currently being used in their probabilistic cost 
analysis. This analysis will present alternatives that may be used to improve the current 
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approach, to more accurately predict the cost of a naval vessel and the risks associated 
with its development. The main focus is on data elicitation, choice of probability 
distributions, the MAIMS principle, and correlations. With the proposed cost analysis 
approach, the management involved in the development and construction of naval vessels 
will have a better tool to reduce program costs, while increasing the probability of 
success for each project. 

E. SCOPE AND METHODOLOGY 

Because of the complexity of the NAVSEA 05C CG(X) model, this thesis focuses 
on the analysis of the electronics portion. The electronics suite is comprised of nine 
different systems and related components. From this analysis, a model is developed to 
provide a more effective method to assess cost risk. Crystal Ball® software is used to 
provide analysis with the use of Monte Carlo simulations and charts to help quantify the 
importance of different risk factors and their link to the overall cost risk. 

A literature review of risk assessment, project risk management, cost analysis, and 
cost modeling is conducted in Chapter II. Data is then analyzed in Chapter III, to 
determine how the NAVSEA CG(X) model was developed, what the assumptions were, 
what distributions were used, and if correlation effects were used. In Chapter IV, 
Crystal Ball® software is used to conduct Monte Carlo simulations on modified models to 
determine if there was any significant change in the cost based on proposed assumptions. 
In these simulations, experiments with different distributions, correlation effects, and 
truncation of the distributions to simulate the MAIMS principle are conducted. Finally, 
Chapter V summarizes the proposed cost risk analysis improvements that were 
investigated in this thesis. 


10 



II. LITERATURE REVIEW 


A. INTRODUCTION 

Cost estimation is a process that has been more closely scrutinized in recent 
history because of the increasing trouble with cost overruns by major programs. The 
Littoral Combat Ship (LCS) is an example of gross cost overruns. The 2008 
United States Government and Accountability Office (GAO) review, in March 2008, 
revealed that the Navy was expecting the first two LCSs to exceed their budgets by more 
than 100 percent, while delivery of the first ship would be at least 18 months later than 
originally projected. These cost and schedule overruns led to the cancelled construction 
of the third and fourth LCSs. Funds originally allocated for construction of the fifth and 
sixth LCSs were allocated to pay for the extensive cost growth experienced by LCSs one 
and two (Littoral Combat Ship Program, 2008). Secretary of the Navy Donald Winter 
states: “ ... we recognize that active oversight and strict cost controls in the early years 
are necessary to ensuring we can deliver these ships to the fleet over the long term” 
(Scully, 2007). Inability to forecast the required cost for future ships and programs will 
not allow the Navy to reach its goal of a 313-ship Fleet by 2020. 

Risk comes in various forms in the business of shipbuilding. Schedule, 
technology, source selection, and requirements are only a few of the factors that are 
associated with uncertainty and risk. The following statements, somewhat sarcastically, 
but with much insight, sum up potential impacts associated with risk in a project: 

• “Any task can be completed in only one-third more time than is currently 
estimated.” - Augustine’s Law of Unmitigated Optimism, Law Number 
XXIII (Augustine, 1997). 

• “The only thing more costly than stretching the schedule of an established 
project is accelerating it, which is itself the most costly action known to 
man.” - Augustine’s Law of Economic Unipolarity, Law Number XXIV 
(Augustine, 1997). 
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• “The process of competitively selecting contractors to perform work is 
based on a system of rewards and penalties, all distributed randomly.” - 
Augustine’s Law of the Phoenix, Law Number XXXIV (Augustine, 1997). 

• “Ninety percent of the time things will turn out worse than you expect. 
The other 10 percent of the time you had no right to expect so much.” - 
Augustine’s Law of Apocalyptic Costing, Law Number XXXVII 
(Augustine, 1997). 

• “One should expect that the expected can be prevented, but the unexpected 
should have been expected.” - Augustine’s Law of Amplification of 
Agony, Law Number XLV (Augustine, 1997). 

• “The sooner you start to fall behind, the more time you will have to catch 
up” (Augustine, 1997). 

B. HISTORY OF COST RISK ASSESSMENT 

Gambling has been a human pastime for millennia and has been practiced at all 
levels of societies. Evidence of gambling can be found in Greek mythology, in which a 
game of “craps” was used to explain what we commonly refer to as the Big Bang. In 
Greek mythology, Zeus, Poseidon, and Hades rolled dice for parts of the universe. Zeus 
won the heavens, Poseidon the seas, and Hades became the master of the underworld 
(Bernstein, 1998). 

Modern-day risk analysis has roots in the Hindu-Arabic numbering system as 
early as seven to eight hundred years ago. The new numbering system allowed writing to 
take the place of movable counters in making calculations. The act of being able to write 
calculations opened mathematics up to a new level. Risk analysis became a more serious 
topic during the Renaissance, in a time when people started to break free from past 
constraints and old beliefs to new challenges. In 1654, Chevalier de Mere challenged 
Blaise Pascal to answer a question that had been posed by Luca Paccioli over two 
hundred years prior. The problem was to determine how to divide the stakes of an 
incomplete game of chance between two different players when one of the players is 
ahead. In seeking out his answer, Pascal sought the help of Pierre de Fermat. Together 
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they came up with a solution to the Paccioli puzzle. This revelation led to the theory of 
probability, the heart of the concept of risk. Through this discovery, people could now 
make decisions and use numbers to forecast events in the future (Bernstein, 1998). In 
1730, Abraham de Moivre discussed the structure of the normal distribution and 
developed the standard deviation (Bernstein, 1998). Both of these concepts are important 
components of modern day methods for conducting risk assessment. 

Historically, cost growth has been associated with many large capital projects, 
especially those that have long construction periods. The trend has been that larger 
budgets and longer build times lead to greater risk and uncertainty. Figure 2 shows a 
history of cost growth over the last 700 years. This chart shows different reasons for 
which each of these major projects came in under, over, or right on cost estimates. An 
interesting note is that the Eiffel Tower project builder was guaranteed the first 20 years 
of revenue (Deegan, 2007b). This fact may have helped to influence the overall cost of 
the project to come in under budget. All of the causes listed in Figure 2 can be applied as 
reasons for problems with the accuracy of cost estimates in current shipbuilding projects. 
Being able to quantify cost variability and uncertainty is one way to help alleviate major 
cost overruns. 


Possible Cause 

Thames 

Tunnel 

Eiffel 

Tower 

Concorde 

Suez 

Canal 

S>xlne>' 

Opera 

House 

Channel 

Tunnel 

Excessive Schedule Pressure 

/ 





✓ 

Changing Requirements 



✓ 


✓ 


Requirements Creep 







Limited Teclmical Specifications 



✓ 




New Technology 

/ 


✓ 


/ 

/ 

hisuflicient Engineenng Experience 

✓ 


/ 

✓ 



Ignoring the Obvious 

✓ 




✓ 


Isnorins Test Results 







Political 


✓ 

/ 

✓ 

✓ 

/ 


•400«o 5*0 4i200*b -2000«o N .\ -180*0 


Figure 2. Historical Cost Growth: Fast 700 Years 

(From: Deegan, 2007b) 
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C. CLASSICAL COST ESTIMATION AND CONTINGENCY PLANNING 

Cost estimation is an analytical effort directed at calculating and predicting the 
cost of a system that involves several techniques such as data collection and analysis and 
risk analysis. It is a quantitative assessment of the most likely costs required to complete 
a project and always involves amounts of uncertainty and risk. The validity and 
significance of a cost estimate relies on the experience and judgment of those assigned 
with making the decisions involved in the estimate. There is no one correct answer when 
making a cost estimate as several factors must be taken into account in order to provide 
the best estimate for the given situation. Associated costs may include labor, material, 
equipment, inflation, and services. Cost estimators interact with many different 
personnel while collecting data for analysis. Consultation with SMEs, engineers, 
schedulers, accountants, statisticians, and mathematicians may occur in the cost 
estimation process (Project Manager’s Body of Knowledge (PMBOK), 2004). 

Classical cost estimation techniques involve using point estimates as the best 
value while also assigning a project contingency as a percentage of the total cost. This 
methodology is changing to the practice of using probability distributions to model cost 
without contingencies. Cost contingencies have been associated with overestimating 
costs to account for the contingency allowances. Program managers are responsible for 
the budget management of their projects and can benefit from using cost estimation 
techniques for use in budget planning and execution (Portny, Mantel, Meredity, Shafer, & 
Sutton, 2008). 

D. PROBABILISTIC COST RISK ANALYSIS 

Modern cost uncertainty analysis started in a field known as military systems 
analysis, which was founded in the 1950s at the RAND Corporation. The military 
systems analysis branch evolved after World War II and became a tool to aid defense 
leadership with decisions for force structure, composition, and theaters of operation. 
Naturally, cost analysis became an important part of the analysis models and decision 
process. Early cost uncertainty analysis focused on the sources, scope, and types of 
uncertainties that influenced the cost of future technologies. Erom 1955 to 1962, the 
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technical papers regarding cost uncertainty did not focus on developing formal 
methodologies to quantify cost uncertainty (Garvey, 2000). In the 1960s, a new focus on 
cost analysis had begun and forms the basis for today’s cost estimation methods. 

Cost estimation uncertainties are present in many elements of cost analysis. They 
originate from inaccuracies in models, misinterpretation of the models, misused cost 
analysis methodologies, economic uncertainties, requirements change, and system 
definition uncertainties. Cost risk analysis is the statistical treatment of the cost 
estimating process that considers the elements of the work breakdown structure (WBS) 
and total system costs as random variables. Cost analysis indicates that there is always 
some degree of risk that a system may not be delivered or meet required objectives at a 
certain funding level. One of the goals of cost analysis is to be able to assign a degree of 
confidence to any particular estimate of a system cost. Another factor to consider when 
conducting cost risk assessment is to analyze what funding level percentage is 
appropriate for the project. 

Most cost estimation is based on historical data obtained from SMEs and 
databases from previously built systems. Basing a cost estimate on past systems can be 
tricky and complicated. Cost analysts have been expected to provide a best estimate of 
cost for various options at milestones and decision points in the acquisition of major 
weapons programs. What precisely is the best estimate? Definitions of the term “best 
estimate” vary with different projects, and with each analyst and program manager. The 
best estimate may be the most likely cost, the median, or the average cost. The term 
“best estimate” implies that other less likely solutions may exist. 

In probabilistic or risk cost estimation, costs of each WBS element are modeled 
with probability distributions and then correlations among these elements are estimated. 
The result of the correlated distributions is statistically summed by using a simulation 
such as Monte Carlo. The simulation results in the total system cost represented by a 
probability distribution. Estimates of statistical parameters, such as the median or other 
pertinent values, can be extracted from the final probability distribution. These values 
are meaningful to program managers and decision makers when establishing 
program budgets. 
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Cost estimation can also be used as a decision aid when program managers need 
to make a decision about the appropriate funding level for a project. Figure 3 shows the 
impact of either budgeting with a value less or greater than the mean. Both decisions 
have possible positive and negative outcomes. It is important for the program manager 
and budget decision makers to have the most complete understanding of the cost analysis 
and consequence to make the best budget decisions. 



Figure 3. Business Rules to Consider: Choosing an acceptable cost risk point requires 

an understanding of consequence (From: Deegan, 2007b). 


Figure 4 is a flowchart that represents the basic steps in estimating probable 
systems cost (Book, 2001). This thesis focuses on the “Probability Distributions, 
Perform Monte Carlo Simulations, Cumulative Distribution, and Read Off Cost 
Percentiles” blocks of this process within the Cost-Risk Analysis branch. 
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Figure 4. Basic Steps in Estimating Probable Systems Cost (From: Book, 2001) 


E. CHAPTER SUMMARY 

There is an ongoing major shift in research and development and complex 
engineering projects from deterministic to probabilistic approaches. Probabilistic cost 
risk analysis provides the proper framework for handling the many different factors that 
contribute to cost uncertainties. It is an important tool for program managers to use in 
making better budget decisions. This is especially significant for the today’s Navy during 
a time of budget hardship. In order for the Navy’s plans for a 313-ship Fleet to become a 
reality, the incorporation of cost risk analysis into the aspects of acquisition and 
shipbuilding is imperative. Cost assessment can help to deliver projects on time, at a 
lower cost, with a higher probability of success. Effective training of personnel involved 
in cost assessment, and continued efforts to improve on existing models will help to more 
fully integrate this way of doing business into the current cost estimating process. 
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III. REVIEW OE THE NAVSEA 05C (COST ENGINEERING AND 
INDUSTRIAL ANALYSIS DIVISION) COST RISK MODEL 


A. INTRODUCTION 

The cost model used for analysis in this thesis was obtained from NAVSEA 05C. 
Mr. Chris Deegan, the former Director of the Naval Sea Systems Command Cost 
Engineering and Industrial Analysis Division, and Mr. Aaron Ratliff, his CG(X) analyst, 
provided the model. The model contains information required to determine the cost of an 
entire CG(X) ship including economic factors such as inflation, profit, and learning 
curves for the complete CG(X) fleet. Because of the complexity of the model, this thesis 
focuses on the electronics cost portion and the effects of data elicitation, probability 
distribution choice, correlation effects, and the MAIMS principle on the probabilistic 
costs associated with technology, design, engineering, production, testing, and integration 
risks for the first vessel. 

B. REVIEW OF CG(X) SHIP CLASS AND MODELS 

The CG(X) ship is designed to be the future Fleet’s replacement for the 
Ticonderoga Class AEGIS Cruisers. New technologies being advocated and presented in 
CG(X) include a stealthier hull form, improved missiles, integrated power systems, and 
advanced radar systems (Navy of the Future, 2008). To lessen costs, the CG(X) was to 
share many common features with DDG 1000. As of July 2008, only two of the 
DDG 1000 ships will be built due to funding constraints. This is a significant reduction 
from the original 32-ship DDG 1000 plan of the 1990s (Cavas, 2008a). These same 
funding constraints place an important emphasis on effective cost and risk modeling for 
CG(X), to help ensure that its acquisition is not put in jeopardy due to unforeseen 
cost issues. 

The NAVSEA CG(X) cost model uses historical data from DDG 1000 expressed 
in FY2005 dollars. It comprises 63 separate worksheets within the main Excel® 
workbook. The data used in this model consists of inputs from SME inquiry, historical 
data from past shipbuilding projects, historical data from shipyard figures, and historical 
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rate data. The NAVSEA 05C CG(X) eost model takes into aeeount all faetors involved 
in eost for every aspeet of the ship ineluding labor rates, material eost, overhead eost, 
planning eost, produetion eost, engineering eost, eonstruetion eost, ehange orders, 
eleetronies eost, hull, meehanioal and engineering eosts, and ordnanee. Figure 5 is the 
Cost Plaeemat of the NAVSEA 05C CG(X) model, which shows how the cost breakdown 
is established. 
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Figure 5. NAVSEA 05C CG(X) Cost Model: Cost Plaeemat showing cost breakdown. 


Figure 6 shows a section of the Mission Systems Risk Assessment worksheet of 
the NAVSEA 05C CG(X) Excel® workbook. This worksheet shows the different cost 
elements of the WBS for the electronics suite of CG(X) as well as the WBS elements for 
the ordnance suite. The elements of the ordnance suite are not considered in this thesis. 
This figure is a snapshot of a single step of the Monte Carlo simulation. Note that 
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Xl(Rec) was generated as part of the thesis using the Exeel® spreadsheets provided by 
NAVSEA 05C. The eolumn “XI (Ree)” represents the random variables eorresponding 
to the reeurring eosts of the first ship of the elass. It eorresponds to the probabilistie 
eosts, whieh inelude uneertainty information. The values for the XI eosts are ealeulated 
using the subsystem cost values provided by the Input Template worksheet shown in 
Eigure 7. The column showing nonrecurring engineering (NRE) costs are costs for only 
the lead ship of the class. The values in the “most likely” column are point estimates 
with the “low” and “high” values as the upper and lower cost bounds of the individual 
electronics suite elements. 

The Naval Surface Warfare Center (NSWC), Dahlgren, Virginia, provided the 
low, most likely, and high values for the system and component costs. These values 
provide the basic input for this thesis. The electronics values and how they are 
determined are the focus of this thesis. 


MAMDJF AoA Lead Ship Production Cost Estimate Risk Anaiysis 

All Costs Reported in Year of award 


X1 (Lead Ship) (TY11$M) 


WBS 

NRE 

XI (Rec) 

% Low 

Low 

Most Likely 

High 

% High 

Distribution 

Ordnance 

$ 

14.70 

$107.84 


$ 

86.30 

$ 

93.77 

$ 165.74 


Extreme 

VLS 

S 

320 

$70 93 

95% 

S 

58 60 

S 

61 68 

$ 101.57 

165% 

Triangular 

GIGS 

$ 

11.500 

S 31.97 

85% 

S 

23 63 ' 

S 

27 80 

S 55 60 

200% 

Extreme 

DLS 

$ 

S 4.93 

95% 

s 

4 07 

s 

4 29 

S 8 58" 

200% 

Extreme 

Electronics 

$ 

160.38 

$ 804.53 


'$ 633.57 

$ 

748.19 

'$910.20 



Radar Suite 

$ 

23.15 

$ 257.49 

85% 

$ 190.32 

$ 

223.90 

$ 257.49 

115% 


X-band 

$ 


95% 

5 

62.30 

s 

65.58 

S 75.41 

115% 

Triangular 

S-band 

s 

22 72 


95% 

S 146.S8 

S 

154.61 

5 177.80 

115% 

Triangular 

Cooling 

s 

0.43 


95% 

S 

3.53 

S 

3.71 

S 4.27 

115% 

Triangular 

Power (included in Ship Total) 

s 

1 82 


95% 

S 

46.17 

S 

48.60 

S 55.89 

115% 

Triangular 

ExComm 

s 

30 21 

S 154 44 

75% 

S 100 72 

s 

134 29 

S 188 0l" 

140% 

Triangular 

TSCE 

s 

3 45 

S 86.61 

75% 

5 

56 49 

s 

75 32 

S 94 15 

125% 

Normal 

lUSW 

s 

5 69 

S 30 38 

95% 

S 

25 09 

s 

2641 

S 27 73 

105% 

Normal 

EW-IW 

s 

21 02 

S 5610 

75% 

S 

36 58' 

s 

48 78 

S 60.97" 

125% 

Normal 

EO-IR 

$ 

S 14.15 

95% 

S 

11 69 

s 

12 31 

S 12 92 

105% 

Normal 

IFF 

$ 

$ 9.65 

95% 

$ 

7 98 

s 

8 40 

S 8 82 

105% 

Normal 

MS EI&T (SS Only) 

s 

60.89 

S 105.19 

80% 

S 

73 18 

s 

91 47 

$ 109.77 

120% 


MS EI&T (CS Only) 

s 

15 98 

S 90 52 

80% 

S 

62 97 

s 

78 71 

S 94.45" 

120% 


ATP 

s 

S 1.38 

95% 

s 

1.31 

s 

f.38 

S 1.45 

105% 

Normal 


Eigure 6. NAVSEA 05C CG(X) Model - Section of Mission Systems Risk Assessment 
worksheet depicting WBS Ordnance and Electronics Suite elements. It captures a 
snapshot of a single step of Monte Carlo simulation run using Crystal Ball®. 
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Figure 7 shows a section of the Input Template worksheet from the NAVSEA 
05C CG(X) model. As indicated by the “Distribution Used” column, NAVSEA 05C uses 
a lognormal distribution for each cost component in the electronics suite. It then 
characterizes each lognormal distribution in terms of the mean and standard deviation 
values given in the Risk Ranges Mission Systems worksheet, a section of which is shown 
in Eigure 8. Analysis of the provided data appears to indicate that NAVSEA 05C 
computes the mean and standard deviation values treating the “low,” “most likely,” and 
“high“ values as a discrete, rather than a triangular, distribution. This calculation of the 
standard deviation and mean for the lognormal distribution has not been confirmed with 
NAVSEA 05C. This approximation provides values significantly different from those 
based on a triangular distribution, as is typically done in today’s de facto probabilistic 
cost analysis. NAVSEA 05C introduces an additional approximation and treats the radar 
suite as a single composite cost, rather than in terms of its individual cost elements. 


Input Template (Solutio 

ns) 

Risk. Factors for Ship X 

Uaits 

V»l«t 

Dis tribe tioa 

Used 

Oistr. Parameter 

1 

Oistr. Parameter 

2 


Mission Sfstems 




VLS 

X 


LeoNorm)! 

120% 

35% 






Mon 

St Deviation 


CIGS 

X 

lOO^i 

LogNormal 

100% 

15% 






Mon 

St Deviation 


DLS 

X 

100% 

LogNorm)! 

102% 

45% 






M«»n 

St Deviation 


Radar Suite 

X 

100^1 

LogNormal 

101% 

17% 






Mon 

St Deviation 


ExComms 

X 

100% 

LogNormal 

10S% 

35% 






Mon 

St Deviation 


SEICcT 

X 

100X 

LogNormal 

100% 

26% 






Mon 

St Deviation 


TSCE 

X 

100% 

LogNormal 

100% 

26% 






Mon 

St Deviation 


lUSV 

X 

100^ 

LogNortnal 

100% 

3% 






Mon 

St Deviation 


EV-IW 

X 

100% 

LogNormal 

100% 

35% 






Mon 

St Deviation 


EO-IR 

X 

100% 

LogNormal 

100% 

16% 






Mon 

St Deviation 


IFF 

X 

100% 

LogNormal 

100% 

13% 






Mon 

St Deviation 










Eigure 7. NAVSEA 05C CG(X) Model- Section of the Input Template worksheet 

depicting the systems analyzed in this thesis. 
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Recurring 



VLS 




Low Most Likeli High 

Average 

Std Deviation 




$ 58.60 $ 61.68 $ 101.57 

74 

24 <—due to bias 



XStd 

^2y. Ky. 20y. 

my. 

my. 



Std Dev 

7.031573242 6.1680467 20.313214 

7 

3 <..... due to estimate variability incl. Correlation 







74 

26 <— due to estimate variability * bias 




35y. 







Enter Estimated Correlations among variables (O-None, .2*some. .5-moderate. 1.0-perfect), positive or negative 



Correlalion 

1 2 3 





1 0.2 0.2 




2 

0.2 1 0.8 




3 

0.2 0.8 1 








EVV-IW 



Lov Most Likeli Hiqk 

Avernqe 

tKd Deeintioi Note: For EW-IW, Single Est. Onlv, assume *~25X 




t 36.58 t 48.78 $ 60.37 

43 

12 <..... due to variance within one estimate 



%Std 

25% 25% 25% 

15% 

25% 



Std D«v 

3.1461001 12.1348 15.2435 

7 

12 <..... due to estimate variability incl. Correlation 








43 

17 <..... due to estimate variability ♦ bias 



35% 






Enter Estimated Correlations among variables fO-None, .2‘some, .5-moderate, 1.0*perfectl, positive or negative 



Correlation 

1 2 3 




1 

1 1 1 




2 

1 1 1 




3 

1 1 1 





EO-IR 



Lov Most Likcli Hiqk 

Avernqe 

Ud DeTiatioi Note: For EO-IR, Single Est. Only, assume ^-25^ 



t 11.63 t 12.31 t 12.32 

12 

1 <..... due to variance within one estimate 



%Std 

15% 15% 15% 

3% 

15% 



S(d D«v 

1.7535267 1.845818 1.33811 

1 2 due td eistimate vafiabilituincLCorrelati&n 





12 

2 < -due (0 vnrinbilitv ♦ bin; 



16% 







EnUr E;titn;iUd Correlation; nmonq vnrinbic; fO-None, .2-;ome. .5-moderntc. 1.0-p«rfectl, pofitivc or negative 



Correlation 

1 2 3 





1 

1 1 1 





2 

1 1 1 





3 

1 1 1 





IFF 



Lov Most Likeli Hiqk 

Avernqe 

Ud Devintion 




$ 7.38 t 8.40 $ 8,82 

8 

0 due to bias 



%Std 

15% 15% 15% 

3% 

12% 



Std Dev 

1.1363313 1.253236 1.32226 

1 

1 <..... due to estimate variability incl. Correlation 







8 

1 due to estimate variability ♦ bias 




13% 






Enter Estimated Correlations among variables fO-None, .2-some, .5-moderate, I.O-perfect), positive or negative 



Correlation 

1 2 3 




1 

1 0.5 0.5 




2 

0.5 1 0.5 




3 

0.5 0.5 1 








Figure 8. NAVSEA 05C CG(X) Model - Risk Ranges Mission Systems worksheet. 
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C. DATA SELECTED FOR FURTHER ANALYSIS 

This thesis uses the low, most likely, and high data values, which are provided by 
the Mission Systems Risk Analysis worksheet shown in Figure 6. The Mission Systems 
Risk Analysis worksheet is one of the 63 worksheets of the CG(X) model. The Xl(Rec) 
value is derived from data contained in other worksheets in the model. The details of the 
methodology of how the Xl(Rec) value is developed is outside the scope of this thesis. 
This thesis then evaluates the impact of different distributions, data elicitation, 
correlations, and the MAIMS principle on the estimated cost. 

The thesis investigates the following probability distributions: 

• Triangular distributions using specified high, most likely, and low values. 

• Lognormal using the mean and standard values from the 
triangular distributions. 

• Two sets of Weibull distributions using the triangular distribution 
parameters as different percentiles to reflect data elicitation. 

• One set of Weibull distributions modified to account for the 
MAIMS principle. 

It also investigates correlation effects using a two-parameter model (Kujawski 
et ah, 2004). 

A discussion of data elicitation practices and methods is presented, and its impact 
on the quality of data is explored. Data elicitation can directly affect the choice of 
distribution used to realistically evaluate cost. Data that is elicited using proven methods 
and/or based on historical data give more realistic and credible predictions than data that 
is simply a best guess. This is where the importance of data elicitation from the SME can 
make a dramatic impact on the quality of data obtained for use in the model. Augustine 
sums up the impact of poor data: 

The weaker the data available upon which to base one’s conclusion, the 

greater the precision which should be quoted in order to give the data 

authenticity. - Augustine’s Law of Definitive Imprecision, Law Number 

XXXV (Augustine, 1997). 
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The final analysis of this thesis involves the investigation of the impact of 
modifying the Weibull distributions to simulate the MAIMS principle, assuming the 
project will spend all money allocated and not have a cost underrun. 

D. CHAPTER SUMMARY 

The NAVSEA 05C’s cost analysis model of CG(X) encompasses all aspects of 
cost for the entire fleet including inflation and profit in a 63 worksheet Excel® workbook. 
The data used in the NAVSEA model were acquired from SME inquiry using three-point 
estimates of high, most likely, and low values. The Navy is placing great emphasis on 
producing the best product for each dollar spent. In order to ensure the continued 
acquisition of CG(X), it is important that realistic cost risk analysis be performed so that 
program managers can make informed decisions. 

Chapter IV presents an approach to improve on the model that NAVSEA 05C has 
provided for CG(X). The focus is strictly on the methodology used in the cost analysis of 
the electronics suite of CG(X), represented in the Mission Systems Risk Assessment 
spreadsheet of the model. 
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IV. REVISED COST RISK ANALYSIS 


A. INTRODUCTION 

This thesis specifically looks at the electronics portion of the CG(X) cost model 
and cost uncertainties associated with engineering and manufacturing of the lead vessel. 
Nine systems make up the electronics suite: 

• Radar suite, which consists of the following subsystems: X-band, S-band, 
Cooling, and Power 

• ExComm - External Communications 

• TSCE - Total Ship Computing Environment 

• lUSW - Integrated Undersea Warfare 

• EW-IW - Electronic Warfare-Information Warfare 

• EO-IR - Electro Optical-Infrared 

• lEE - Identification, Eriend or Eoe 

• MS EI&T (SS Only) - Mission Systems Engineering, Integration, and 
Testing (Ship Systems Only) 

• MS EI&T (CS Only) - Mission Systems Engineering, Integration, and 
Testing (Combat Systems Only) 

The electronics suite cost is determined with the following two equations that 
treat each of the cost elements as a random variable (RV). The costs in bold represent 
composite of the costs in regular font, which Crystal Ball refers to as forecasts 
and assumptions: 

COST(Electronics Suite) = COST(Radar Suite) -t COST(ExComm) -t 

COST(TSCE) -t COST(IUSW) -t COST(EW-IW) -t COST(EO-IR) -t COST(IEE) 

-t COST(MS EI&T (SS Only)) -t COST(MS EI&T (CS Only)), 
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Cost(Radar Suite) = Cost(X-band) + Cost(S-band) + Cost(Cooling) + 

Cost(Power). 

The steps of analysis for the CG(X) model are: 

1. Analyze the cost factors used by NAVSEA 05C to develop the electronics 
suite cost. 

2. Analyze the PDFs used for the electronics cost elements. 

3. Identify what data elicitation methods were employed. 

4. Determine if correlation factors were used in the cost analysis. 

5. Develop cost factors to be modeled for cost realism. 

6. Decide which PDFs to use for greater fidelity. 

7. Develop an improved cost risk model that includes realistic correlation 
factors; credible PDFs, including MAIMS influences; and SME biases. 

B. REVIEW DEVELOPMENT OF COST FACTORS 

The cost factors used in the NAVSEA05C CG(X) cost model for the electronics 
suite in this thesis include data elicitation methods, PDF choice, correlation effects, and 
MAIMS Principle effects. This research does not pursue the other elements of cost such 
as labor, material, or inflation rates that NAVSEA 05C models. 

1. Data Elicitation Methods 

The data elicitation methods used by NAVSEA 05C cost analysts are not well 
documented. It is clear that the engineering and expert judgment of SMEs is heavily 
relied on for the assessment of uncertain cost elements associated with new designs. This 
is an area where the use of improved methods can dramatically improve the quality of 
data that is used in the computation of the cost risk model. Subjective assessments to 
obtain data have been identified as a critical source of uncertainty in probabilistic risk 
analyses (Keeney & von Winterfeld, 1991). Kujawski et al. (2004) discuss the use of the 
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DFA method for data elicitation and how this ties in with distribution choice, to provide 
the most realistic cost assessment. 

DFA has been found to provide one of the most consistent and least bias-prone 
methods for eliciting uncertain quantities (Alpert & Raiffa, 1982). In their research, 
people were asked to consider uncertain quantities by providing values in terms of 
percentiles or fractiles. The findings indicated: 

• There is a systematic bias toward overconfidence in estimates. The 
subjective probability distributions were too narrow. Usually, 33% 
instead of 50% of the actual values fell within the 0.25 to 0.75 fractiles. 

• Extreme value judgment is even worse. Twenty percent, rather than 2%, 
of the actual values fell outside the 0.01 and 0.99 fractiles. 

• What is the meaning of minimum and maximum values? Defining these 
terms, so that they are universal, is difficult. 

Kujawski et al. (2004) further suggest using experts to provide the 10th, 50th, and 
90th percentile values, as these may be easier to assess than extreme values of maximum 
and minimum. They recommend avoiding asking for extreme values, abstract values 
such as the mean or standard deviation, or other specific distribution functions. If the 
analyst does not fully understand the background of the questions being asked to obtain 
data, or if he does not fully understand the behavior of the system and associated data, 
obtaining discrete values will be near impossible. 

Education also plays an important role in the quality of the data provided by the 
SMEs for analysis. The understanding of bias and its role in affecting data elicitation is 
important. In a presentation to the Navy Cost Analysis Symposium, Eields and Popp 
(2007) stress the importance of several lessons learned on risk. One of the most 
interesting of these lessons learned is the importance of training. They indicate that 
although NAVSEA and its technical community have a broad cross section of 
educational backgrounds and experience, this does not mean that everyone has 
experience in simulation and statistics. The SME for a particular electronics suite 
component is probably not an expert in probability and statistics, and because of this. 
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tends to give biased answers to the cost analysis. The distributions formed from the 
biased data have been found to be particularly narrow and centered on a given point 
estimate, while the extreme values are very rarely taken into account, for reasons 
described above. 

Education of the SMEs while conducting data elicitation is important, so that the 
experts have a better understanding of what data is required and how it is going to be 
utilized. This training needs to be continually refreshed due to the high turnover rate of 
personnel, whether they be military or civilian, and also because of improving methods 
for cost analysis. An adequate training plan for both the cost analysts and the SMEs 
providing data will ultimately result in better data acquisition for cost analysis. 

In this thesis, the use of DEA is simulated though the use of Weibull probability 
distributions because no new data elicitation was conducted. The differences between the 
distributions using identical values for 10th, 50th, and 90th percentiles versus the 20th, 
50th, and 80th percentiles illustrates data elicitation that is optimistic versus pessimistic. 
The resulting cost associated with each of the two distributions shows how dramatic the 
effects of slightly different parameters can have on the estimated cost. 

2. Choice and Development of Probability Distribution Functions 

Kujawski et al. (2004) emphasize the importance of realistically modeling cost 
uncertainties through the appropriate choice of probability distribution by meeting the 
following criteria: 

• Capable of fitting three arbitrary percentiles. 

• A finite lower range. 

• An infinite upper range with reasonable behavior. 

• Physically meaningful and easy to estimate parameters. 

Three types of PDEs are developed and modeled for this thesis, with the goal of 
finding a realistic and flexible probability distribution. Uncertainty for each cost element 
in the cost model is represented using the same type of PDE with different parameters 
(based on NAVSEA data). Eirst, a triangular PDE that uses low, most likely, and high 
values for its parameters is developed. A lognormal PDE that uses the mean and standard 
deviation from the triangular PDE as its parameters is the second distribution. The third 
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PDF is a three-parameter Weibull distribution based on the low, most likely, and high 
values of the triangular PDF provided by NAVSEA 05C. The low and high values are 
ealeulated by multiplying the low and high pereentages obtained with the most likely 
value shown in Figure 6. Two Weibull distributions are ereated. One of the Weibull 
distributions uses the data as input for the 10th, 50th, and 90th, percentiles, while the 
other is more pessimistic and uses these values for the 20th, 50th, and 80th percentiles. 
The data for these parameters is taken from the Mission Systems Risk Assessment 
worksheet (see Figure 6). For consistency, the triangular distribution is used to determine 
the 50th percentile. The low, 50th percentile, and high values are substituted for the 20th, 
50th, and 80th percentiles and for the 10th, 50th, and 90th, for each three-parameter 
Weibull distribution, respectively. Table 2 shows the values that were used for the 
parameters of each probability distribution function for the different electronics 
suite elements. Crystal Ball® determines the three-parameter Weibull distribution by 
using the specified three percentiles. For example, the parameters of the Weibull (10%, 
50%, 90%) distribution for the X-Band element are: location = 60.02, scale = 9.21, and 
shape = 1.61. For thoroughness, I verified that Crystal Ball® generates identical 
distributions using three percentiles and the three standard Weibull parameters. 



A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

L 

1 


Distributions 










2 



Triangular 


Lognormal Triangular 

Weibull (10, SO, 90) 

Weibull (20, SO, 80) 

3 

WBS 

Low 

Most Likely 

High 

Mean 

Standard 

Deviation 

10% 

50% 

90% 

20% 

50% 

80% 

4 

X-band 

S62 30 

$65 58 

$7541 

$67.76 

$2.78 

$62 30 

$67.36 

$7541 

$62.30 

$67.36 

$7541 

5 

S-band 

$146,88 

$154.61 

$177,80 

$159.83 

$6.56 

$146 88 

$158.90 

$177 80 

$146 88 

$158.90 

$177 80 

6 

Cooling 

$3.53 

$3.71 

$4.27 

$3.84 

$0.16 

$3,53 

$3.82 

$4.27 

$3.53 

$3.82 

$4.27 

7 

Power (included in Ship Total) 

$46.17 

$48 60 

$55 89 

$50.24 

$2.06 

$4617 

$49.95 

$55 89 

$4617 

$49.95 

$55 89 

8 

ExComm 

$100.72 

$134.29 

$188 01 

$141.19 

$18.12 

$100.72 

$139.89 

$188 01 

$100.72 

$139.89 

$188.01 

9 

TSCE 

$5649 

$75.32 

$9415 

$75.29 

$7.67 

$5649 

$75.33 

$9415 

$5649 

$75.33 

$9415 

10 

lUSW 

$25 09 

$2641 

$27 73 

$26.42 

$0.54 

$25.09 

$26.41 

$27 73 

$25 09 

$26.41 

$27 73 

11 

EW-IW 

$36 58 

$48 78 

$60 97 

$48.68 

$5.03 

$36 58 

$48.63 

$60 97 

$36 58 

$48.63 

$60 97 

12 

EO-IR 

$1169 

$1231 

$12.92 

$12.31 

$0.25 

$1169 

$12.31 

$12.92 

$1169 

$12.31 

$12,92 

13 

IFF 

$7 98 

$840 

$8 82 

$8.40 

$0.17 

$7 98 

$8.40 

$8 82 

$7 98 

$8.40 

$8 82 

14 

MS EI&T(SS Only) 

$7318 

$9147 

$109 77 

$91.61 

$7.55 

$7318 

$78.85 

$109 77 

$7318 

$78.85 

$109 77 

15 

MS EI&T(CS Only) 

$62,97 

$78.71 

$94.45 

$78.77 

$6.45 

$62.97 

$91.67 

$9445 

$62.97 

$91.67 

$9445 


Table 2. Parameters used in the probability distribution functions for the different 

electronics suite elements. 
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a. Triangular Probability Distribution Function 

The parameters used to develop the triangular PDF are the low, most 
likely, and high values from the Mission Systems Risk Assessment worksheet (Figure 6). 
The determination of the high and low percentages for cost values in the Mission Systems 
Risk Assessment worksheet were figures given to NAVSEA 05C cost analysts by SMEs 
from the NSWC Dahlgren. These percentages are based on historical database values 
and inquiry of the SME for an opinion about what they thought the low and high values 
would be, based on the most likely values obtained from the historical databases. In this 
case, data elicitation plays a big part in the reliability of the data used in the model, which 
is be described in more depth in Section IV.B.l. 

The triangular distribution is not a good predictor of high and low costs 
because it uses the low and high values as extreme values for the end points. There is no 
allowance for costs above or below the input values. It has been argued that a triangular 
distribution can lead to either underestimates or overestimates. Graves (2001) states that 
underestimates are likely due to the finite upper limit of the distribution. Moran (1999) 
believes that overestimates happen because of the distribution’s inability to portray the 
expert’s confidence level of achieving the most likely value and/or knowledge of the 
shape of the distribution. The triangular distribution is assigned a very low score for 
criteria (i) and (iii) (Kujawski et ah, 2004) and is not the chosen distribution to represent 
cost in the model for this thesis. 

b. Lognormal Probability Distribution Function 

The lognormal PDE is created with the mean and standard deviation 
parameters taken from the triangular distribution. Characteristics of a lognormal 
distribution include being positively skewed with a limitless upper bound and known 
lower bound. This distribution is assigned an acceptable score for criteria (iii), but a low 
score for (i), due to the always positively skewed nature of the distribution. The 
lognormal distribution results in a cost profile that closely follows with the triangular 
distribution, and is one of the narrowest profiles modeled. A lognormal PDE has been 
associated with providing unreasonably high probabilities at high values, due to the 
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relatively slow falloff to the right. For this reason, it gets an acceptable score for the 
criteria (ill), but scores low on the criteria (i) because of its always positively skewed 
characteristic (Kujawski et al., 2004). 

c. Weibull Probability Distribution Function (10%, 50%, 90%) 

The three-parameter Weibull distribution is characterized by being flexible 
and able to assume a wide variety of shapes, while also being open-ended. Because of its 
flexible profile and ability to mimic other distributions, it scores high on all criteria. 

This thesis models one of the three-parameter Weibull distributions with 
the 10th, 50th, and 90th percentile values for cost. The parameters of 10th, 50th, and 
90th percentiles are chosen to simulate a cost environment that allows for some cost 
flexibility on the upper and lower limits rather than making them extreme as in the 
triangular distribution. Although this 10% change on either side of the distribution seems 
large, it actually represents a fairly optimistic assessment of cost. This model is best for a 
situation in which the data obtained for the model is very reliable. 

d. Weibull Probability Distribution Function (20%, 50%, 80%) 

The three-parameter Weibull distribution using the 20th, 50th, and 80th 
percentiles for distribution parameters is intended to correct or account for the overly 
optimistic biases discussed in Section 2 above. Systems that are new and untested have a 
certain amount of uncertainty inherent in their acquisition, and most cost assessments 
made on their components are based on past history if components are being reproduced, 
or a best estimate for new systems and their components. SMEs are naturally optimistic 
about their systems and have been shown to give cost estimates that are overconfident, 
resulting in probability distributions that do not accurately reflect the possible range of 
costs (Kujawski et al., 2004). 

Much data for the CG(X) electronics suite is the result of inquiry of SMEs 
and because of this, the Weibull distribution using 20%, 50%, and 80% parameters is 
chosen to model costs for the electronics suite components in this thesis. 
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e. Cost Comparisons with the Different Probability Distributions 

Figure 9 is the Excel® overlay created with Crystal Ball® that shows of a 
10000-run Monte Carlo simulation for the triangular, lognormal, Weibull (10%, 50%, 
90%) and Weibull (20%, 50%, 80%) distributions, representing the electronics suite cost 
of the CG(X). Figure 10 is the cumulative probability distribution derived from the PDF 
shown in Figure 9. The triangular and lognormal distributions are very similar in both 
the probability distribution and cumulative frequency functions, which is expected. Since 
the lognormal distribution uses the mean and standard deviation from the triangular 
distribution as its parameters, the end result should be very similar. Both the triangular 
and lognormal functions show a very distinct peak and sharp falloff at both the lower and 
upper bounds. This behavior does not realistically model the electronics suite cost 
because of the sharp peak with sharp falloff. 


Delta Elicitation and Distribtuion Selection Effects 
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Figure 9. CG(X) Crystal Ball® Analysis, 10000 Runs showing the Effects of 
Distribution Choices on the Cost probability distributions. 


34 
























Data Elicitation and Distribution Effects 
for CG(X) Electronics Cost 
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Figure 10. CG(X) Crystal Ball® Analysis, 10000 Runs showing the Effeets of 
Distribution Choices on the cost Cumulative Distribution Functions. 

The Weibull (10%, 50%, 90%) distribution shows a broader cost range for the 
given probability brackets. The tapering lower and upper bounds in comparison to the 
triangular and lognormal distribution represent a more likely cost outcome. The Weibull 
(20%, 50%, 80%) shows an even larger cost range, which makes sense because this 
distribution is supposed to model a more pessimistic view of cost. Both of these 
distributions are associated with higher costs as the probability of the cost increases. It is 
important to note the difference between the optimistic and pessimistic Weibull 
distributions in Figure 10. For each, the cost increases with an increase in probability, 
but it is clear that the model’s results indicate that the cost risk is significantly higher 
using the pessimistic Weibull distribution. 

3. Correlation Effects 

Correlation effects are potentially important in modeling appropriate cost 
relationships between different elements of systems and are not conducted enough in 
current cost analysis models (Book, 2001). Trends with correlation tend to lean toward 
perfect correlation because of simplicity. Perfect correlation does help to widen the range 
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of outputs in the distribution functions, but this may not be an accurate or reliable 
representation. Reasonable correlation coefficients may provide more realistic and 
credible estimates of project costs, rather than assuming perfect or zero correlation. 
Assessing correlation coefficients is a difficult problem. A need exists for the 
investigation and development of a realistic and practical model to account for 
interrelationships between cost elements. 

Two types of correlations are modeled in this thesis: 

• Correlations among cost elements within the radar suite. The radar suite 
includes elements of X-band, S-band, Cooling, and Power. Dependencies 
among these components are mainly from subsystem characteristics such 
as complexity. 

• Correlations among cost elements in the entire electronics suite. 
Dependencies among these cost elements occur from the programmatic 
and organizational considerations common to all cost elements that are a 
part of the same project (Kujawski et ah, 2004). 

There are two types of correlations: Pearson and Spearman. Pearson correlation 
coefficient determines the degree of linearity between two random variables, while 
Spearman rank order correlation coefficients measure monotonicity. Correlation among 
cost elements in the electronics suite is modeled with the use of the Correlation Matrix 
function in Crystal Ball®. Crystal Ball® uses rank correlations to correlate assumptions. 
This means that the values are not changed, but they are rearranged to produce the 
desired correlation. Rank correlation eliminates the need to explicitly model the 
dependence between the cost elements. Garvey (2000) advocates the use of Pearson’s 
correlation. However, given the limited information, rank correlations offer the 
advantage of accounting for correlations independent of explicit distribution and 
dependency models. The use of Monte Carlo simulations generates the full PDF rather 
than simply expected value and variance. 

This thesis uses three sets of two correlation coefficients to model the correlation 
between the radar suite elements and the rest of the electronics suite components. The 
first set models the distributions with correlation coefficients of 0.5 for the radar suite 
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elements and 0.5 for the entire eleetronics suite elements. The seeond set of eorrelation 
eoeffieients is 0.5 and 0.2. The third set uses eorrelation eoeffieients of 0.8 and 0.5. 
Figure 11 shows the eorrelation ehart used for the (0.5, 0.2) eorrelation. The 
eorresponding Weibull (20%, 50%, 80%) distribution model that the eorrelation ehart in 
Figure 11 is applied to is shown in Figure 12. 

To further explain the correlation chart in Figure 11, please consider the cell 
showing the correlation factor for the E7 (Weibull 20, 50, 80) and E8 (Weibull 20, 50, 
80) cost elements. As indicated by Eigure 12, the cells E7 and E8 correspond to the 
X-band and S-band systems, respectively. Both are components of the radar suite and 
therefore are likely to have strong dependencies. Weibull (20%, 50%, 80%) denotes that 
the elements are modeled using three-parameter Weibull distribution given by the 20th, 
50th, and 80th percentiles. The correlation cell is determined by the intersection of the 
corresponding row and column for the two elements. The value of this cell is 0.50. This 
indicates that the correlation between the elements X-band and S-band of the radar suite 
are correlated by a factor of 0.5. 
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Eigure 11. CG(X) Crystal Ball® Analysis, 10000 Runs, Correlation (0.5, 0.2) chart for 

Weibull (20%, 50%, 80%) distribution model. 
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Figure 12. CG(X) Crystal Ball® Analysis, 10,000 Runs, Weibull (20%, 50%, 80%) 

distribution model with (0.5, 0.2) correlation. 

Figure 13 is an overlay of the different probability distributions for the electronics 
suite cost, produced by using the following three different combinations of correlation 
coefficients for the radar suite and electronics suite: 


• Correlation coefficients of 0.5 among the radar suite components and 0.2 
between the different electronic suite components. 

• Correlation coefficients of 0.5 among the radar suite components and 0.5 
between the different electronic suite components. 

• Correlation coefficients of 0.8 among the radar suite components and 0.5 
between the different electronic suite components. 

As discussed above, positive correlations give rise to broader distributions, which 
reflect higher uncertainty. The no correlation PDF in Figure 13 is the same as the no 
correlation shown in Figure 9. They do not appear to be the same due to the difference in 
scale and also due to the fact that they are from separate Monte Carlo simulations. 
Although the Monte Carlo simulations will give similar results for each run, they will not 
be identical. Analysis of the effects of the different correlation coefficients is discussed 
further in Section IV.C. 
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Figure 13. CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Eleetronics Suite eost 
based on different correlation effects in cumulative probability form. 


4. MAIMS Principle Effects 

The MAIMS principle is modeled in this thesis by using the three-parameter 
Weibull (20%, 50%, 80%) distribution function and predetermined percentile points for 
the MAIMS set points. By implementing MAIMS into the distribution function, any 
value less than the money allocation point is equal to the money allocation value. The 
percentage parameters used for MAIMS are the 50th percentile or median, the mean, and 
the 80* percentile funding levels. A spike, or delta function, is observed in the MAIMS 
modified distributions at the money allocation points. These money allocation points 
correspond to the budget allocated to the WBS cost elements by the project manager. 

The MAIMS modified functions are modeled by using the following equation: 

If Distribution Value < X, then X, else Distribution Value 
By using this equation, the value of the MAIMS modified distribution will never be less 
than the value X. 

Figure 14 is a screenshot of a Monte Carlo simulation run with the MAIMS 
principle. The formula bar shows the equation used to simulate the MAIMS principle for 
cell G7. This equation was used for the three different MAIMS scenarios modeled. 
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Figure 15 shows the resulting PDFs obtained from the model in Figure 14. Further 
analysis of the effects of the MAIMS principle on cost is discussed more in depth later in 
this chapter, in Section IV.D.3. 
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Figure 14. CG(X) Crystal Ball®Analysis, 10000 Runs, Electronics Suite costs including 

the MAIMS principle. 
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Figure 15. CG(X) Crystal Ball®Analysis, 10000 Runs, Electronics Suite costs including 

the MAIMS principle, PDEs. 
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5. 


Other Factors 


Uncertainty ranges for the low, most likely, and high values are modeled using 
triangular distribution functions and a Monte Carlo simulation. The purpose of doing this 
simulation is to ensure that any uncertainty effects from the low, most likely, and high 
values could be accounted for in the model. By doing this, the low, most likely, and high 
values become assumption cells as defined by Crystal Ball®. These assumptions are used 
for the input parameters of the “Distribution Value” cell, which gives the cost value of 
the particular element. The results indicate that the total cost distribution has values 
essentially indistinguishable from the original. This is because sampling the distributions 
of the individual low, most likely, and high values produces values that are very similar 
to using the original values. Because the modeling did not significantly change the result 
of the input parameters to the “Distribution Cell,” uncertainty distribution functions for 
the low, most likely, and high values were not used in the model for this thesis. 
Figure 16 shows the model used for this experiment. 
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Figure 16. CG(X) Crystal Ball® Analysis, 10000 Runs, Modeling Low, Most Likely, and 

High values: Low Value Distribution. 
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c. 


DESCRIPTION OF THE CRYSTAL BALL MODEL 


This section describes the modeling and analysis of the cost factors discussed in 
Section IV.B using Crystal Ball®. 

The model for the CG(X) cost analysis for this thesis is developed by using the 
template provided by NAVSEA 05C for their CG(X) cost model. Specifically, it utili z es 
the low, most likely, and high cost values for the triangular distribution from the Mission 
Systems Risk Assessment worksheet. The low, high, and 50th percentile values are used 
as the input parameters for the three-parameter Weibull probability distributions, Weibull 
(10%, 50%, 90%) and Weibull (20%, 50%, 80%). The lognormal distributions used the 
means and standard deviations of the corresponding triangular distributions. These cost 
element distributions were created and simulations were run with Crystal Ball® software. 
Figure 17 is a screenshot of a single step run of a Monte Carlo simulation for a given set 
of PDFs. 
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Figure 17. CG(X) Crystal Ball® Analysis, 10000 Runs, Screenshot of CG(X) single step 

Monte Carlo Simulation. 

The left column of the model shows the components of the electronics suite. The 
next three columns contain the data that was acquired from the NAVSEA 05C Excel® 
worksheets. The “Distribution Value” column contains green boxes, which correspond to 

the assumptions for the cost elements. The assumption cells are defined as probability 
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distributions and contain the random variables for the eost elements. The blue boxes in 
the “Distribution Value” eolumn are the forecasts that refer the sum of the assumption 
boxes as follows: (1) The radar suite eost is the probabilistie sum of the X-band radar, 
S-band radar, eooling, and power eosts; and (2) the eleetronics eost is the probabilistie 
sum of the cost of the radar suite, the ExComm, TCSE, lUSW, EW-IW, EO-IR, lEE, and 
MS EI&T costs. The MAIMS columns represent the values for the simulation that 
correspond to a MAIMS modified distribution. The MAIMS 50% column represents a 
distribution truncated at the 50th percentile, while the MAIMS mean is truncated at the 
mean value, and the MAIMS 80% is the distribution modified to start at the 80th 
pereentile value of the baseline distributions. 

Eigure 18 shows a snapshot of how the distribution parameters are input into the 
model for a triangular distribution. The low, most likely, and high eost values are entered 
in the appropriate blocks in the dialog sereen called “Define Assumption” to develop the 
distribution that will be represented by the appropriate cell. In this case, the eell being 
defined is E7. 
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Eigure 18. CG(X) Crystal Ball® Analysis, 10000 Runs, Screenshot of the 

Development of Triangular Distribution Eunetion. 
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Figures 19, 20, and 21 are sereenshots of how distributions are ereated for the 
lognormal, Weibull (10%, 50%, 90%) and Weibull (20%, 50%, 80%) distributions. For 
Figure 18, the lognormal distribution, and the input parameters of mean and standard 
deviation are obtained from the results of the triangular distribution funetion from 
Figure 17. The three-parameter Weibull distributions shown in Figures 19 and 20 use the 
low values as either the 10th or 20th pereentiles and the high values as the 80th or 90th 
percentiles, depending on the function and the 50th percentile for the third parameter. 



Note that for Figures 20 and 21, the distributions represented for cell E7 (X-band) is 
highly skewed. This is due to the fact that the low and most likely values are very 
similar, while the high values are significantly higher, skewing the resulting distribution. 


Figure 19. CG(X) Crystal Ball® Analysis, 10000 Runs, Screenshot of the Development 

of Lognormal Distribution Function. 
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Figure 20. CG(X) Crystal Ball® Analysis, 10000 Runs, Screenshot of the 

Development of Weibull (10%, 50%, 90%) Distribution Function. 



Figure 21. CG(X) Crystal Ball® Analysis, 10000 Runs, Screenshot of Development of 

Weibull (20%, 50%, 80%) Distribution Function. 

Correlation distributions are created by using the Correlation Matrix function of 
Crystal Ball®. Specific correlation coefficients are entered into the distribution functions 
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of assumption cells in the dialog window under the “Correlate” button. When the model 
is run with eorrelation eoeffieients set for specifie distribution funetions, the results are 
now eorrelated. Figure 22 is a sereenshot showing how eorrelation eoeffieients are added 
to an assumption eell. 



Figure 22. CG(X) Crystal Ball® Analysis, 10000 Runs, Screenshot of Development of 

Model using Correlation Coefficients. 


After the models are developed and run 10000 times, overlays are created to 
graphieally show the results. These overlays are deseribed in more detail in 
Seetion IV.D. 


D. RESULTS 


1. Effects of Distribution Choice on Cost Forecast 

The first distributions modeled were the single eleetronics suite elements with 
different distributions. For the purpose of this thesis, the element ExComm is chosen for 
this explanation. Figure 23 shows the eumulative frequeney distribution of the different 
modeled distributions for the ExComm element. The triangular and lognormal 
distributions show similar characteristics, which is expected since the lognormal 
distribution uses parameters taken from the triangular distribution (mean and standard 
deviations). Both Weibull distributions show expected charaeteristics. The Weibull 
(20%, 50%, 80%) definitely indicates a more pessimistic cost forecast because as the 
cumulative probability increases, the cost increases more signifieantly than for the 
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Weibull (10%, 50%, 90%) distribution. This overlay indicates that the choice of 
distribution used in modeling does play a significant part in results obtained for cost. The 
three-parameter Weibull distributions represent a more realistic cost outcome for high 
risk components. Weibull distributions allow for modeling of highly complex 
distributions using DFA, while triangular distributions have a more restrictive shape, 
making it difficult to fit three arbitrary percentiles for the low, most likely, and high 
values (Kujawski et ah, 2004). 



8(J 125 1/1 215 252 3U2 

Cost, $M 

Figure 23. CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics Suite 
element ExComm cumulative frequency distributions for different 
probability distributions. 

The probability distribution functions shown in the overlay in Eigure 24 illustrate 
expected behaviors for the ExComm PDEs. Both the triangular and lognormal 
distributions are narrow because the triangular distribution upper and lower bounds do 
not allow for infinite upper cost ranges. The sum of the Weibull (20%, 50%, 80%) 
distributions shows a more pessimistic behavior in comparison to the sum of the Weibull 
(10%, 50%, 90%) distributions. 
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Figure 24. CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics Suite 

element ExComm with different PDEs. 

Once all the individual electronics suite elements are modeled, they are summed 
up probabilistically in the main worksheet in Excel to obtain the entire electronics suite 
cost. The simulation selects a random value from each of the element distributions then 
adds them to create one data point for the total cost. This is repeated 10000 times to 
create the total cost distribution. When all the distribution functions (assumption cells in 
the model) of the electronics suite elements are probabilistically summed, the resulting 
cost is illustrated in the overlay shown in Eigure 25. All four distributions have the 
appearance of a normal distribution consistent with the Central Eimit Theorem 
(Garvey, 2000). 

The lognormal and triangular distribution functions give rise to relatively narrow 
total cost distributions, consistent with the finite ranges of the contributing triangular 
distributions and the modeling of the lognormal distributions using the corresponding 
mean and standard deviation values. The Weibull (10%, 50%, 90%)-based cost 
distribution shows narrower behavior for cost range than the Weibull (20%, 50%, 80%)- 
based distribution. The Weibull (20%, 50%, 80%)-based distribution allows for more 
uncertainty in data elicitation from SMEs. Weibull distributions not only show higher 
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probabilities of cost overruns, but also higher probabilities of cost underruns. These 
underruns reflect the assumption of 10% and 20% as the low value parameter for the 
distribution, rather than using it as the minimum value. Figure 26 shows the same data as 
Figure 25, except that it is in the cumulative probability form. 


Data Elicitation and Distribtuion Selection Effects 
for CG(X) Electronics Cost 



575 660 74 5 831 916 993 

Cost, $M 


Figure 25. CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics Suite Cost 

based on different distribution selections. 


Data Elicitation and Distribution Effects 
for CG(X) Electronics Cost 



Figure 26. CC(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics Suite cost 
based on different distribution selections in cumulative probability form. 
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Effects of Correlation on Cost 


As discussed in Section IV.B.l, two types of correlations are modeled in this 
thesis: (1) Correlations among cost elements within the radar suite, and (2) correlations 
among cost elements in the entire electronics suite. This thesis uses the following three 
different correlation coefficient factors to show the correlation between the radar suite 
elements and the rest of the electronics suite components: 

• Radar suite correlation coefficient = 0.5, electronics suite correlation 
coefficient = 0.5 

• Radar suite correlation coefficient = 0.5, electronics suite correlation 
coefficient = 0.2 

• Radar suite correlation coefficient = 0.8, electronics suite correlation 
coefficient = 0.5. 

The choice to use the values listed above is to simulate an environment that is not 
a perfectly correlated or no-correlation situation. The (0.5, 0.5) correlation assumes that 
there is an equal correlation relationship between the subcomponents of the radar suite 
and the elements of the electronics suite. The (0.5, 0.2) correlation illustrates the effects 
of having a stronger correlation between the elements of the electronics suite than 
between elements of the entire electronics suite. The (0.8, 0.5) correlation show the 
impact of a stronger correlation between components of one system than between 
different systems. These correlation coefficients represent a limited set of parameters for 
investigation in this thesis. Further research in the determination of appropriate 
correlation coefficients and their effect is needed to provide a more complete analysis. 

The impact of correlation effects is seen in Figures 27 and 28. These overlays 
show the different distributions that are a result of a 10000-run Monte Carlo simulation 
for the correlated distributions in cumulative distribution form (Figure 27) and PDF form 
(Figure 28). The blue PDF is the reference Weibull (20%, 50%, 80%) distribution with 
no correlation effects. This distribution has the narrowest cost range when compared 
with the correlated distributions. The cost ranges of the Weibull distribution increases as 
the correlation factors increase. Also, in the cumulative probability distribution shown in 

Figure 27, all of the distributions intersect at the mean value for the cost. 
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CGX Electronics Cost 
Correlation Effects 



Figure 27. Figure 27: CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics 

Suite Cost showing the impact of different correlation effects. 


CGX Electronics Cost 
Correlation Effects 



Figure 28. Figure 28: CC(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics 
Suite cost based on different correlation effects in cumulative probability form. 


As expected, the (0.5, 0.2) correlation being the smallest has the least effect on the 
total cost distribution. It is interesting to note that the distribution resulting from the 
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(0.5, 0.5) correlation does not differ much from the (0.8, 0.5) distribution, but that both of 
these correlations have a more significant effect than the (0.5, 0.2) correlation. This 
indicates that a change in the the correlation factor for the radar suite from 0.5 to 0.8 is 
not as significant as a change in the correlation factor from 0.2 to 0.5 for the different 
components of the entire electronics suite. These results suggest that the correlation 
effects are important for probability values midpoint between the mean and the extremes, 
but there is little difference for values beyond 0.5. The results in Figure 28 are consistent 
with theoretical predictions of positive correlation effects in that the total cost becomes 
broader than for uncorrelated total cost (Kujawski et al., 2004). Further investigations are 
recommended to quantify correlation effects. 

3. MAIMS Effects on Cost 

The MAIMS modified cumulative probability and density density distributions 
for the electronics suite cost are shown in Figures 29 and 30. Characteristics of the 
MAIMS modified PDF is that they will never have a value less than the chosen value of 
modification. So, for the MAIMS 50th percentile modified distribution in Figure 29, the 
distribution has no value less than the 50th percentile baseline cost level. In Figure 30, 
the spikes or delta functions normally associated with the individual MAIMS 
distributions are not seen as they are modulated when summed. 
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CGX Electronics Cost 
Weibull (20. 50. 80) MAIMS Effects 



Figure 29. CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics Suite cost 

showing the MAIMS effects in cumulative probability form. 


CGX Electronics Cost 
Weibull (20. 50. 80) MAIMS Effects 



637 694 752 809 866 918 


Cost, $M 


Figure 30. CG(X) Crystal Ball® Analysis, 10000 Runs, Overlay of Electronics Suite cost 

showing the MAIMS effects in PDF form. 
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It is important to note the significant rise in cost curve that occurs as the MAIMS 
modification value increases. The mean value of the distribtion increases as the funding 
level increases, and this is very clear in Figure 29. The curve representing the MAIMS 
80% distribution shows how the budget is always high when comparing it to the MAIMS 
50% or MAIMS mean distribution. This effect is because once money has been allocated 
to a WBS element, it is almost never seen in cost savings as underruns, because cost 
account managers never return money to the project. Any remaining money from one 
WBS is subsequently spent on a different existing WBS that has cost overruns. 

These simulations can be considered with other cost factors in making program 
management decisions regarding budgets. Funding projects at a level too low to cover 
costs will lead to cost overruns, while funding at a level that is too high leads to money 
not being recouped as savings later. Allocating reasonable budgets is the goal. 

E. CHAPTER SUMMARY 

The research for this thesis is based on the NAVSEA 05C CG(X) model provided 
by Mr. Chris Deegan and his CG(X) analysts. The CG(X) model encompasses all factors 
considered for cost of the entire program, including labor rates, material cost, overhead 
cost, planning cost, and other factors. Because of the complexity of the model and the 
numerous factors to consider, one portion of the model was chosen for analysis. The 
Electronic Suite and its nine elements are specifically targeted as the focus for analysis. 

The steps used in the analysis of the CG(X) model are: 

1. Analyze the cost factors used by NAVSEA 05C to develop the 
electronics cost. 

2. Analyze the PDEs used for the electronics cost elements. 

3. Identify what data elicitation methods were employed. 

4. Determine if correlation factors were used in the cost analysis. 

5. Develop cost factors to be modeled in a new model. 

6. Decide which PDEs to use in the new model. 

7. Develop a new cost model using correlation factors, chosen PDEs, and 
MAIMS influenced distributions. 
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Identified cost factors include NAVSEA 05C’s probability distribution choice, 
method used for developing the low, most likely, and high cost values for the electronics 
suite elements, data elicitation methods, and correlation effects. This thesis explores the 
methodology in choosing different probability distribution functions and their 
applicability to the model. Specifically, triangular, lognormal, and two variations of the 
three-parameter Weibull distribution are considered. 

Methods of data elicitation are explored and the use of a DFA method is 
recommended for future use, although the research in this thesis did not involve data 
acquisition. To simulate the use of a DFA methodology, Weibull distributions are 
employed to account for uncertainty associated with SME estimation of data. A Weibull 
(10%, 50%, 90%) distribution is used to simulate a more optimistic view of the 
uncertainty of data, while a Weibull (20%, 50%, 80%) distribution models a more 
pessimistic, but probably more realistic, view of the uncertainty associated with data from 
the SMEs. 

Two types of correlation effects are considered and modeled in this thesis. The 
first is the correlation between subcomponents of the radar suite and the other is the 
correlation between the elements of the electronics suite. The radar suite is one of the 
elements that make up the electronics suite. Analysis shows that a more significant effect 
is experienced with higher correlation between the elements of the electronics suite than 
between the subcomponents of the radar suite. 

MAIMS modified probability distributions are modeled to show the significance 
of budget allocation level. These distributions are truncated at the baseline budget with a 
delta function at the baseline. This is based on the principle that once a budget is 
allocated, money is almost never seen in the form of cost under runs as the project 
progresses. As the MAIMS modification value increases, overall distribution cost rises 
with increasing probability of success. 
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V. CONCLUSIONS 


A. SUMMARY 

This thesis begins by exploring the definitions of risk and how it applies to the 
guidance set forth by current Navy leadership. Admiral Gary Roughead, Chief of Naval 
Operations, states that, “We manage risk” (Roughead, 2007). The need to develop 
effective acquisition and shipbuilding methods to successfully deliver an “affordable 
future fleet” (McCoy, 2008) is imperative if the Navy is to meet the goal of a 313-ship 
Navy by 2020. Cost risk analysis is one tool of many that can be used to help attain 
this goal. 

This thesis then proceeds to examine the probabilistic cost analysis approach that 
NAVSEA 05C currently uses to predict new naval vessel construction costs and to 
develop a method that better predicts the ultimate cost risk. Cost factors analyzed in this 
thesis include the effect of data elicitation, distribution choice, the impact of the MAIMS 
principle, and the effect of correlation factors. Data elicitation and MAIMS have 
significant impact. Correlation effects vanish at the minimum, mean, and maximum 
values. PDF selection has a small impact as long as the distributions fit the three 
specified percentiles. 

The model provided by NAVSEA 05C encompasses all aspects of the ship’s cost 
and only the nine elements of the electronics suite were chosen for analysis in this thesis. 
Using data obtained from SMEs for low, most likely, and high cost values, experiments 
were conducted for the noted cost factors in the Excel® Monte Carlo simulation add-in 
Crystal Ball®. 

Triangular, lognormal, Weibull (10%, 50%, 90%) and Weibull (20%, 50%, 80%) 
distributions are modeled and simulated to show the impact that each distribution can 
have on budget considerations for program managers. Both the triangular and lognormal 
distributions show narrow cost ranges when compared to the Weibull distribution cost 
range. The Weibull (10%, 50%, 90%) represents a more optimistic distribution than the 
more pessimistic Weibull (20%, 50%, 80%) distribution. The Weibull (20%, 50%, 80%) 
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distribution accounts for the optimism bias commonly associated with SMEs. Data 
elicitation effects are modeled through the use of the Weibull distributions. 

Correlation among cost elements in the electronics suite is modeled with the use 
of the Correlation Matrix function in Crystal Ball®. This thesis uses three sets of two 
correlation coefficients to model the correlation between the radar suite elements and the 
rest of the electronics suite components. The results suggest that the correlation effects 
are important for probability values midpoint between the mean and the extremes, but 
there is little difference for values beyond 0.5. Further investigations are recommended 
to quantify correlation effects. 

MAIMS principle modified distributions are modeled with the 50th percentile 
cost value, mean, and 80th percentile cost value to show the impact of funding at these 
different levels. The MAIMS principle is based the observation that for a given budget, 
any money allocated is considered money spent. Very rarely are cost underruns 
experienced on a project once the budget has been allocated. The MAIMS modified 
distributions in this thesis show the impact of either under-funding a budget or over- 
funding. Under-funding leads to cost overruns and over-funding leads to an overall 
higher cost, since money allocated is unlikely to be recouped. 

B. RECOMMENDATIONS AND AREAS FOR FURTHER RESEARCH 

The analysis conducted in this model is only a starting point for improvements in 
the area of cost analysis for naval vessels. Although the methodology used in this thesis 
provides a framework for obtaining more accurate predictions of cost than those in use 
with current probabilistic cost analysis, more work is required to develop a more 
complete and tested model. Recommendations for future research in the area of 
probabilistic cost analysis for shipbuilding include: 

• Use of the DFA method to obtain data for cost assessment. Recommend 
eliciting data from SMEs at the 10th, 50th, and 90th percentiles, at a 
minimum, for relatively optimistic view of the data quality, and at the 
20th, 50th, and 80th percentiles if a more pessimistic view of the quality 
of data is present. Take into consideration the overconfidence of estimates 
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provided by experts in their field and use this knowledge when calibrating 
data for analysis. 

• Select flexible and realistic probability distribution functions for 
cost analysis. Create probability distribution functions from historical data 
and adjust for expected differences in new programs. 

• Incorporate the use of correlation among cost elements of a system. Aim 
to use a range of correlation coefficients that is realistic. A reasonable 
range for correlation coefficients is between 0.3-0.6, with some room for 
variation. Overly optimistic correlation coefficients that assume 
independence and overly pessimistic correlation coefficients that assume 
perfect correlation rarely exist in real data. 

• Use the “Money Allocated is Money Spent” (MAIMS) principle to model 
budget management behavior. The MAIMS function will not allow the 
system cost to be a lesser amount than the budgeted cost baseline. 

• Investigate further capabilities available with advanced modeling software 
such as Crystal Ball® or ©Risk. 

• Incorporate systems engineering methodologies and thinking into the 
development of probabilistic cost analysis. Kujawski et al. (2004) state 
that this is the single greatest challenge to the development and use of 
improved cost models. 

Continuing with the development of improved cost models is an important step in 
helping the Navy to ensure the successful acquisition of the 313-ship Navy it desires. 
Improved cost models can give project managers the ability to develop more realistic and 
successful plans for their projects, while enabling them to make better budget decisions. 
The cost analysis methodology presented in this thesis can serve as a starting point for 
further advanced research in this area that can be used by different programs across 
the Navy. 
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